Information processing system

ABSTRACT

An information processing system includes an attribute-information acquisition unit and an identification unit. The attribute-information acquisition unit acquires an attribute of an object that is present in a real space in accordance with information regarding the object an object image of which is captured. The identification unit identifies an image that is a virtual image associated with the attribute acquired by the attribute-information acquisition unit, the image being to be combined with the object image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority under 35 USC 119 fromJapanese Patent Application No. 2016-218417 filed Nov. 8, 2016.

BACKGROUND Technical Field

The present invention relates to information processing systems.

SUMMARY

According to an aspect of the invention, there is provided aninformation processing system including an attribute-informationacquisition unit and an identification unit. The attribute-informationacquisition unit acquires an attribute of an object that is present in areal space in accordance with information regarding the object an objectimage of which is captured. The identification unit identifies an imagethat is a virtual image associated with the attribute acquired by theattribute-information acquisition unit, the image being to be combinedwith the object image.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the present invention will be described indetail based on the following figures, wherein:

FIG. 1 is a diagram illustrating an example overall configuration of aninformation processing system according to the exemplary embodiments;

FIG. 2 is a diagram illustrating an example hardware configuration of anoperation terminal according to the exemplary embodiments;

FIG. 3 is a block diagram illustrating an example functionalconfiguration of an operation terminal according to Exemplary Embodiment1;

FIG. 4 is a table illustrating an example of a virtual-image database(DB) according to Exemplary Embodiment 1;

FIG. 5 is a flowchart illustrating an example of steps of a processperformed by the operation terminal according to Exemplary Embodiment 1;

FIGS. 6A, 6B, and 6C are diagrams for explaining an example of a seriesof steps performed by the operation terminal according to ExemplaryEmbodiment 1;

FIG. 7 is a flowchart illustrating a different example of the stepsperformed by the operation terminal according to Exemplary Embodiment 1;

FIG. 8 is a block diagram illustrating an example functionalconfiguration of an operation terminal according to Exemplary Embodiment2;

FIG. 9 is a table illustrating an example of a virtual-image DBaccording to Exemplary Embodiment 2;

FIG. 10 is a flowchart illustrating an example of steps of a processperformed by the operation terminal according to Exemplary Embodiment 2;

FIGS. 11A and 11B are diagrams for explaining an example of a series ofsteps performed by the operation terminal according to ExemplaryEmbodiment 2;

FIG. 12 is a block diagram illustrating an example functionalconfiguration of an operation terminal according to Exemplary Embodiment3;

FIG. 13 is a table illustrating an example of a virtual-image DBaccording to Exemplary Embodiment 3;

FIG. 14 is a flowchart illustrating an example of steps of a processperformed by the operation terminal according to Exemplary Embodiment 3;and

FIGS. 15A and 15B are diagrams for explaining an example of a series ofsteps performed by the operation terminal according to ExemplaryEmbodiment 3.

DETAILED DESCRIPTION System Configuration

Hereinafter, exemplary embodiments of the present invention will bedescribed in detail with reference to the attached drawings.

First, the overall configuration of an information processing system 1according to the exemplary embodiments will be described. FIG. 1 is adiagram illustrating an example overall configuration of the informationprocessing system 1 according to each exemplary embodiment. Asillustrated in FIG. 1, the information processing system 1 includes anoperation terminal 100 and a management server 200 that are connected toeach other through a network 300.

The operation terminal 100 is a computer having a so-called augmentedreality (AR) function. Examples of the operation terminal 100 include amobile game machine, a mobile information terminal (such as a smartphoneor a tablet terminal), and a personal computer (PC). A wearable computersuch as wearable computer glasses is also usable as the operationterminal 100. The operation terminal 100 runs various programs such asapplications in accordance with a user operation and executes a processfor adding virtual space information to real space information, aprocess for incorporating the real space information into a virtualspace, and other processes. Note that real space is space existing inreality, while virtual space does not exist in reality but is spacevirtually operable with the operation terminal 100.

The management server 200 is a computer that provides the operationterminal 100 with various pieces of information. Examples of themanagement server 200 include a PC and a workstation. The managementserver 200 provides the operation terminal 100 with various pieces ofdata to be used, for example, in a virtual space.

The network 300 is a communication medium used for informationcommunication between the operation terminal 100 and the managementserver 200 and is, for example, the Internet, a public network, or alocal area network (LAN).

Hardware Configuration of Operation Terminal

The hardware configuration of the operation terminal 100 according tothe exemplary embodiment will be described. FIG. 2 is a diagramillustrating an example hardware configuration of the operation terminal100 according to the exemplary embodiment.

As illustrated in FIG. 2, the operation terminal 100 includes a centralprocessing unit (CPU) 101 that is an arithmetic unit, a main memory 102that is a memory, and a magnetic disk device 103. The CPU 101 runs anoperating system (OS) and various programs such as applications andthereby implements various functions of the operation terminal 100. Themain memory 102 serves as a memory area where the various programs, dataused for running the programs, and the like are stored. Further, themagnetic disk device 103 serves as a memory area where data to be inputto the various programs, data output from the various programs, and thelike are stored.

The operation terminal 100 also includes a communication unit 104 thatis an interface for communicating with an external apparatus, a display105 that includes a video memory, a display, and other components andthat displays a screen, and an operation unit 106 that is an inputdevice operable by a user. The operation unit 106 may be any inputdevice, and examples of the operation unit 106 include an operationbutton, a keyboard, and a mouse. The display 105 and the operation unit106 may be integrally formed by using a touch panel or the like thatdisplays a screen and that thereby receives an operation from the user.The operation terminal 100 further includes an imaging unit 107 such asa camera that captures an image of a subject to acquire image data for astill image or a video, and a sound detection unit 108 such as amicrophone that detects a sound outside the operation terminal 100.

Note that FIG. 2 merely exemplifies the hardware configuration of theoperation terminal 100 preferable to apply the exemplary embodiment, andthe configuration to implement the exemplary embodiment is not limitedto the configuration illustrated in FIG. 2.

Exemplary Embodiment 1 Functional Configuration of Operation Terminal

The functional configuration of an operation terminal 100 according toExemplary Embodiment 1 will be described. In Exemplary Embodiment 1, theoperation terminal 100 acquires a data image of an image captured byusing the imaging unit 107 in the real space through a user operation(hereinafter, an image based on image data and captured in the realspace is referred to as a real data image). The operation terminal 100divides the acquired real data image into regions corresponding toobjects. Further, the operation terminal 100 identifies the attribute ofeach region (that is, the attribute of each object). The operationterminal 100 combines a real data image of a region with an image invirtual space based on image data (hereinafter, referred to as a virtualdata image) associated with the attribute of the region.

FIG. 3 is a block diagram illustrating an example functionalconfiguration of the operation terminal 100 according to ExemplaryEmbodiment 1. The operation terminal 100 according to this exemplaryembodiment includes a real-image acquisition unit 111, a region divisionunit 112, a region-attribute identification unit 113, a virtual-imagestorage unit 114, a virtual-image acquisition unit 115, and a combiningunit 116.

The real-image acquisition unit 111 acquires a real data image based onimage data regarding an image captured in the real space by using theimaging unit 107.

The region division unit 112 divides the real data image into regionscorresponding to the objects on the basis of a feature amount extractedfrom the real data image acquired by the real-image acquisition unit111. An existing technique is usable for the region division. Forexample, an existing region division algorithm (such as a regionextension method or graph cuts) is used. Examples of the used featureamount include pixel values of pixels in an image (such as brightness orcolor information) and the frequency of an image (spatial frequency).

The region-attribute identification unit 113 identifies the attribute ofeach region divided by the region division unit 112. The attribute mayalso be identified by using an existing technique.

For example, the region-attribute identification unit 113 compares thefeature amount of the region divided by the region division unit 112with reference feature amounts predetermined for the respectiveattributes. The region-attribute identification unit 113 identifies, asthe attribute of the region, the attribute of a region having areference feature amount closest to the feature amount of the region.

Alternatively, for example, the region-attribute identification unit 113may identify the attribute of the region by using the machine learningin such a manner as to extract the pattern of an image having theattribute and a determination rule therefor. In this case, for example,the user prepares multiple images having the respective attributes asimages for learning. The region-attribute identification unit 113extracts the feature amount from each prepared image for learning andperforms learning by associating the extracted feature amount with theattribute of the image. The region-attribute identification unit 113performs the learning as described above and thereby identifies, as theattribute of the region, the attribute of the image for learning havinga feature amount closest to the feature amount of the region.

The virtual-image storage unit 114 stores therein a database(hereinafter, referred to as a virtual-image DB) in which virtual dataimages corresponding to pieces of image data to be used in virtual spaceare specified. The virtual-image DB is an example of an association, andeach virtual data image is associated with an attribute. The attributerepresents a location where an object indicated by the correspondingvirtual data image is present in the real space. More specifically, forexample, a virtual data image of a bird is associated with the attribute“Sky” of the sky where the bird is present in the real space. Inaddition, for example, a virtual data image of a deer is associated withthe attribute “Mountain” of a mountain where the deer is present in thereal space.

The virtual-image acquisition unit 115 acquires a virtual data imagefrom the virtual-image storage unit 114 as a data image to be combinedwith a real data image. The virtual-image acquisition unit 115 firstselects one or more regions in accordance with a predetermined rule fromamong the regions (objects) of the real data image divided by the regiondivision unit 112. The virtual-image acquisition unit 115 acquires avirtual data image associated with the attribute of each selected regionfrom the virtual-image storage unit 114.

Note that examples of the predetermined rule include selecting the oneor more regions in accordance with the priority or order assigned inadvance to the attributes and selecting the one or more regionsrandomly.

The combining unit 116 combines the real data image of a region with avirtual data image associated with the attribute of the region. Thecombining unit 116 performs control to display, on the display 105, adata image obtained by the combination (hereinafter, referred to as acomposite data image).

In this exemplary embodiment, the real-image acquisition unit 111 isused as an example of an image acquisition unit. The region-attributeidentification unit 113 is used as an example of anattribute-information acquisition unit. The virtual-image acquisitionunit 115 is used as an example of an identification unit. The combiningunit 116 and the display 105 are each used as an example of a display.The virtual-image storage unit 114 is used as an example of a memory.

Note that the functional units included in the operation terminal 100illustrated in FIG. 3 are implemented in cooperation between softwareand hardware resources. Specifically, to implement the operationterminal 100 by using the hardware configuration illustrated in FIG. 2,for example, the OS and application programs stored in the magnetic diskdevice 103 are loaded in the main memory 102 and run by the CPU 101, andthe functions such as the real-image acquisition unit 111, the regiondivision unit 112, the region-attribute identification unit 113, thevirtual-image acquisition unit 115, and the combining unit 116 arethereby implemented. The virtual-image storage unit 114 is implementedby a memory such as the magnetic disk device 103. Virtual-image DB

The virtual-image DB will be described. FIG. 4 is a table illustratingan example of the virtual-image DB according to Exemplary Embodiment 1.

As “Attribute”, attributes respectively associated with virtual dataimages in advance are illustrated. FIG. 2 illustrates a bird as avirtual data image associated with, for example, the attribute “Sky” anda deer as a virtual data image associated with, for example, theattribute “Mountain”. Each attribute may be associated with multiplevirtual data images. The example in FIG. 2 illustrates a fish and awhale as virtual data images associated with the attribute “Sea”.

Steps of Process Performed by Operation Terminal

Steps of a process performed by the operation terminal 100 according tothis exemplary embodiment will be described. FIG. 5 is a flowchartillustrating an example of the steps of the process performed by theoperation terminal 100 according to Exemplary Embodiment 1.

First, when the user operates the operation terminal 100 and captures animage of a subject by using the imaging unit 107, the real-imageacquisition unit 111 acquires a real data image captured by using theimaging unit 107 (step S101). The region division unit 112 divides thereal data image into regions on the basis of feature amounts extractedfrom the real data image (step S102). The region-attributeidentification unit 113 identifies the attribute of each region (object)resulting from the division performed by the region division unit 112(step S103).

The virtual-image acquisition unit 115 selects one or more regions inaccordance with the predetermined rule from among the regions of thereal data image (step S104). The virtual-image acquisition unit 115acquires, from the virtual-image storage unit 114, one or more virtualdata images respectively associated with one or more attributes of theselected one or more regions (step S105). The combining unit 116combines the real data image of each of the selected one or more regionswith the corresponding virtual data image acquired by the virtual-imageacquisition unit 115 and thereby generates a composite data image (stepS106). The combining unit 116 displays the composite data image on thedisplay 105 (step S107). The process is then terminated.

More specifically, for example, in step S104, the virtual-imageacquisition unit 115 selects a sky image and a mountain image. In stepS105, the virtual-image acquisition unit 115 acquires a virtual dataimage of a bird and a virtual data image of a deer as virtual dataimages respectively associated with the attributes of the regions. Inthis case, in step S106, the combining unit 116 combines the sky imagewith the virtual data image of a bird and combines the mountain imagewith the virtual data image of a deer. The combining unit 116 therebygenerates a composite data image.

Here, a case where the user records a video by using the imaging unit107 will particularly be described. While a video is being recorded, asubject recorded by the user changes over time. Accordingly, the processillustrated in FIG. 5 is repeated, for example, every period of timecorresponding to one frame (for example, about every 17 msec in the caseof 60 fps). In step S104, the same region as the previously acquiredregion (or a region including the acquired region) is acquiredrepeatedly. Also in step S105, the same virtual data image as thepreviously acquired virtual data image is acquired repeatedly. Eachvirtual data image is repeatedly combined for the video.

A Series of Steps Performed by Operation Terminal

A series of steps performed by the operation terminal 100 according tothis exemplary embodiment will be described. FIGS. 6A to 6C are diagramsfor explaining an example of the series of steps performed by theoperation terminal 100 according to Exemplary Embodiment 1.

FIG. 6A is a diagram illustrating an example of a real data image 11Acaptured by using the imaging unit 107. For example, the user operatesthe operation terminal 100 and captures an image of a subject by usingthe imaging unit 107. As the result, the real-image acquisition unit 111acquires the real data image 11A illustrated in FIG. 6A as a data imagecaptured by using the imaging unit 107. The real data image 11Aillustrated in FIG. 6A is an image of the sky and mountains captured bythe user. The real data image 11A includes a data image 11B and a dataimage 11C that are a sky image and a mountain image, respectively.

After the real-image acquisition unit 111 acquires the real data image11A, the region division unit 112 divides the real data image 11A intoregions on the basis of feature amounts extracted from the real dataimage 11A. The region division unit 112 divides the real data image 11Ainto the data image 11B and the data image 11C. The region-attributeidentification unit 113 identifies the attributes of the divided dataimage 11B and the divided data image 11C, respectively. Morespecifically, the region-attribute identification unit 113 compares, forexample, the feature amount of the data image 11B with the referencefeature amount predetermined on a per-attribute basis and therebyidentifies the attribute of the data image 11B. The attribute of thedata image 11B and the attribute of the data image 11C are hereinidentified as the attribute “Sky” and the attribute “Mountain”,respectively.

The virtual-image acquisition unit 115 selects one or more regions inaccordance with the predetermined rule from among the regions of thereal data image divided by the region division unit 112. In the exampleillustrated in FIG. 6A, the virtual-image acquisition unit 115 selectsthe data image 11B that is a region having the attribute “Sky”. Thevirtual-image acquisition unit 115 subsequently acquires, from thevirtual-image storage unit 114, the virtual data image of a birdassociated with the attribute “Sky”. Subsequently, the combining unit116 combines the data image 11B having the attribute “Sky” with thevirtual data image of a bird and thereby generates a composite dataimage.

FIG. 6B is a diagram illustrating an example of a composite data image11D. In the composite data image 11D illustrated in FIG. 6B, the dataimage 11B having the attribute “Sky” is combined with a virtual dataimage 11E of a bird as described above.

Note that, it is conceivable that a virtual data image is combined witha real data image, for example, without determining the attribute of aregion of the real data image, that is, on the basis of informationregarding the location in the real space that is acquirable from, forexample, a global positioning system (GPS). In this case, the virtualdata image is combined on the basis of the information regarding thelocation in the real space regardless of a subject in the real space animage of which is captured. Accordingly, as illustrated in FIG. 6C, thedata image 11C having the attribute “Mountain” might be combined withthe virtual data image 11E of a bird. In this case, the bird is presentin the sky in the real space but is displayed as if the bird werepresent in the mountain. The result is an unnatural composite causingthe user to experience an unusual sensation and is not accompanied by afeeling of actually being present.

In contrast, in this exemplary embodiment, the combining unit 116combines a virtual data image on the basis of the attribute of a regionof a real data image. The result is a natural composite that does notcause the user to experience an unusual sensation and is thusaccompanied by a feeling of actually being present.

In addition, if the user records a video by using the imaging unit 107,the location and the size of the sky and mountain images in frames ofthe video change over time. Accordingly, for example, if thevirtual-image acquisition unit 115 acquires the virtual data image 11Eof a bird for the frames, the combining unit 116 combines the dataimages 11B of the sky that change over time in the respective frameswith the virtual data image 11E of a bird. As an additional explanation,for example, if the video recorded by the user has a frame that does notinclude the sky and that corresponds to a certain period of time, thevirtual data image 11E of a bird is not combined with the real dataimage 11A in the frame corresponding to the period of time.

Different Example of Steps Performed by Operation Terminal

A different example of the steps performed by the operation terminal 100will be described. In the aforementioned example, the virtual-imageacquisition unit 115 selects one or more regions of a real data image,and the combining unit 116 combines a real data image of each regionwith a virtual data image. In contrast, in the different example, thevirtual-image acquisition unit 115 acquires a virtual data image(predetermined data image) in accordance with a predetermined rule. Thecombining unit 116 then identifies the attribute of the acquired virtualdata image on the basis of the identified attribute and determines aregion (location) of the virtual data image to be combined in the realdata image. Note that in this exemplary embodiment, the combining unit116 is used as an example of a location identification unit.

FIG. 7 is a flowchart illustrating the different example of the stepsperformed by the operation terminal 100 according to ExemplaryEmbodiment 1.

Since steps S201 to S203 are the same as steps S101 to S103 in FIG. 5,explanation thereof is herein omitted. After the region-attributeidentification unit 113 identifies the attribute of each region of thereal data image in step S203, the virtual-image acquisition unit 115acquires a virtual data image from the virtual-image storage unit 114 inaccordance with a predetermined rule (step S204). Examples of thepredetermined rule include acquiring a virtual data image in accordancewith the priority or order assigned in advance to the virtual data imageand acquiring the virtual data image randomly.

The combining unit 116 identifies the attribute associated with thevirtual data image acquired by the virtual-image acquisition unit 115 onthe basis of information stored in the virtual-image storage unit 114.The combining unit 116 judges whether a region having the identifiedattribute is present in the real data image (step S205). If thecombining unit 116 does not judge that a region having the identifiedattribute is present (No in step S205), the process is terminatedwithout combining the virtual data image.

In contrast, if the combining unit 116 judges that a region having theidentified attribute is present (Yes in step S205), the combining unit116 combines a real data image of the region with the virtual data imageand thereby generates a composite data image (step S206). As anadditional explanation, steps S205 and S206 may be regarded as a step ofidentifying a location for combining a virtual data image with the realdata image on the basis of the attribute of the virtual data image. Thecombining unit 116 displays the composite data image on the display 105(step S207). The process is then terminated.

More specifically, for example, in step S204, the virtual-imageacquisition unit 115 acquires a virtual data image of a bird. In stepS205, the combining unit 116 identifies, as “Sky”, the attributeassociated with the virtual data image of a bird. If a region having theattribute “Sky” is present (Yes in step S205), the combining unit 116combines a sky image with the virtual data image of a bird and therebygenerates a composite data image. The process is then terminated.

Note that in step S204, the virtual-image acquisition unit 115 mayacquire multiple virtual data images. In this case, steps S205 and S206are performed for each virtual data image. Specifically, it is judgedwhether a region having the attribute of the virtual data image ispresent, and if the region is present, the virtual data image iscombined with the real data image.

In this exemplary embodiment as described above, the operation terminal100 combines the real data image captured by using the imaging unit 107,in a region of the real data image, with a virtual data image associatedwith the attribute of the region. Accordingly, the result is a naturalcomposition that does not cause the user to experience an unusualsensation and is thus accompanied by a feeling of actually being presentcompared with, for example, a configuration in which combination isperformed without taking the attribute of a region of a real data imageand the attribute of a virtual data image into consideration.

Exemplary Embodiment 2 Functional Configuration of Operation Terminal

The functional configuration of an operation terminal 100 according toExemplary Embodiment 2 will be described. In Exemplary Embodiment 1, theoperation terminal 100 combines a real data image with a virtual dataimage and thereby generates a composite data image. In contrast, inExemplary Embodiment 2, the operation terminal 100 identifies theattribute of an object in the real data image and performs a process ofincorporating information regarding the object into a virtual space byusing a virtual data image associated with the attribute.

FIG. 8 is a block diagram illustrating an example functionalconfiguration of the operation terminal 100 according to ExemplaryEmbodiment 2. The operation terminal 100 according to this exemplaryembodiment includes a real-image acquisition unit 121, an objectidentification unit 122, a virtual-image storage unit 123, aperformance-information acquisition unit 124, and a virtual-imageincorporation unit 125.

The real-image acquisition unit 121 acquires a real data image based onimage data regarding an image captured in the real space by using theimaging unit 107.

The object identification unit 122 identifies the attribute of an objecton the basis of the image of a predetermined object included in the realdata image acquired by the real-image acquisition unit 121. To identifythe attribute, an existing technique is usable. For example, the useractually captures images of various objects and thereby prepares piecesof image data regarding the objects. The object identification unit 122performs image processing such as pattern matching on the acquired realdata image and thereby identifies the attribute of the object.

The virtual-image storage unit 123 stores therein a virtual-image DBthat is a database in which virtual data images corresponding to piecesof image data to be used in virtual space are specified. In thevirtual-image DB, virtual data images corresponding to predeterminedobjects are specified on a per-predetermined-object-attribute basis.Further, on a per-predetermined-object-attribute basis, data imagesacquired, for example, by actually capturing images of the objects arealso stored. More specifically, for example, if a racing game is playedin the virtual space, the virtual-image DB stores therein virtual dataimages on a per-motor-cycle-accessory-attribute basis, such as for amuffler, a tire, and a handle bar. In addition, on aper-motor-cycle-accessory-attribute basis, data images acquired, forexample, by actually capturing images of the motor cycle accessories arealso stored.

The performance-information acquisition unit 124 acquires informationregarding the performance of the object the attribute of which isidentified by the object identification unit 122 (hereinafter, referredto as performance information). The performance-information acquisitionunit 124 acquires object performance information from an externalapparatus such as the management server 200 through the network 300.More specifically, the performance-information acquisition unit 124searches for the object, for example, through the Internet and therebyacquires the object performance information.

The virtual-image incorporation unit 125 executes a process forincorporating the object performance information acquired by theperformance-information acquisition unit 124 and a virtual data imagecorresponding to the object in the virtual space. In the incorporationprocess, the virtual-image incorporation unit 125 acquires, from thevirtual-image storage unit 123, the virtual data image corresponding tothe object the attribute of which is identified by the objectidentification unit 122, in other words, the virtual data imageassociated with the attribute of the object. The virtual-imageincorporation unit 125 incorporates, into the virtual space, theperformance information and the virtual data image in association witheach other. The incorporation into the virtual space causes the virtualdata image to function as an image of the object having the performanceindicated by the performance information in the virtual space.

More specifically, for example, if a racing game is to be played, theobject identification unit 122 identifies a muffler as the attribute ofan object. In this case, the performance-information acquisition unit124 searches for the muffler through the Internet and acquires theperformance information regarding the muffler. The virtual-imageincorporation unit 125 acquires a virtual data image corresponding tothe muffler from the virtual-image storage unit 123. The virtual-imageincorporation unit 125 incorporates, into the racing game, theperformance information and the virtual data image in association witheach other. The incorporation into the racing game enables the user touse, in the racing game, the muffler having the performance indicated bythe performance information.

In this exemplary embodiment, the performance-information acquisitionunit 124 is used as an example of a performance-information acquisitionunit. The virtual-image incorporation unit 125 is used as an example ofeach of an identification unit and an incorporation unit.

Note that like the functional units included in the operation terminal100 illustrated in FIG. 3, the functional units included in theoperation terminal 100 illustrated in FIG. 8 are implemented incooperation between the software and hardware resources. Specifically,to implement the operation terminal 100 by using the hardwareconfiguration illustrated in FIG. 2, the OS and application programsstored in the magnetic disk device 103 are loaded in the main memory 102and run by the CPU 101, and the functions such as the real-imageacquisition unit 121, the object identification unit 122, theperformance-information acquisition unit 124, and the virtual-imageincorporation unit 125 are thereby implemented. The virtual-imagestorage unit 123 is implemented by a memory such as the magnetic diskdevice 103.

Virtual-Omage DB

The virtual-image DB will be described. FIG. 9 is a table illustratingan example of the virtual-image DB according to Exemplary Embodiment 2.

As “Object attribute”, attributes of the predetermined objects areillustrated. In the example in FIG. 9, “Muffler from Company A”,“Muffler from Company B”, and “Tire from Company C” are illustrated asmotor cycle accessories. As “Captured data image”, data images acquiredby actually capturing images of objects are illustrated. As “Virtualdata image”, data images for a case where objects are used in thevirtual space such as the racing game are illustrated.

Steps of Process Performed by Operation Terminal

Steps of a process performed by the operation terminal 100 according tothis exemplary embodiment will be described. FIG. 10 is a flowchartillustrating an example of the steps of the process performed by theoperation terminal 100 according to Exemplary Embodiment 2.

When the user operates the operation terminal 100 and captures an imageof a subject by using the imaging unit 107, the real-image acquisitionunit 121 acquires a real data image captured by using the imaging unit107 (step S301). The object identification unit 122 judges whether thereal data image has an attribute of a predetermined object on the basisof information stored in the virtual-image storage unit 123 (step S302).If the object identification unit 122 does not judge that the real dataimage has an attribute of a predetermined object (No in step S302), theprocess is terminated.

In contrast, if the object identification unit 122 judges that the realdata image has an attribute of a predetermined object (Yes in stepS302), the performance-information acquisition unit 124 acquiresperformance information corresponding to the predetermined object (stepS303). The virtual-image incorporation unit 125 acquires a virtual dataimage associated with the predetermined object from the virtual-imagestorage unit 123 (step S304). The virtual-image incorporation unit 125incorporates, into the virtual space, the performance information andthe virtual data image in association with each other (step S305). Theprocess is then terminated.

A Series of Steps Performed by Operation Terminal

A series of steps performed by the operation terminal 100 according tothis exemplary embodiment will be described. FIGS. 11A and 11B arediagrams for explaining an example of the series of steps performed bythe operation terminal 100 according to Exemplary Embodiment 2. A casewhere a racing game is played in the virtual space will be described asan example.

FIG. 11A is a diagram illustrating an example of a real data image 21Acaptured by using the imaging unit 107. For example, the user operatesthe operation terminal 100 and captures an image of a subject by usingthe imaging unit 107. As the result, the real-image acquisition unit 111acquires the real data image 21A illustrated in FIG. 11A as a data imagecaptured by using the imaging unit 107. The real data image 21Aillustrated in FIG. 11A is an image of the muffler of a motor cyclecaptured by the user and includes a data image 21B as a muffler image.

After the real-image acquisition unit 121 acquires the real data image21A, the object identification unit 122 judges whether the real dataimage 21A has an attribute of a predetermined object on the basis of theinformation stored in the virtual-image storage unit 123. For example,if the data image 21B is the same as (or similar to) a captured image ofa muffler from Company A illustrated in FIG. 9, the objectidentification unit 122 judges that the real data image 21A has anattribute of a predetermined object on the basis of pattern matching orthe like.

The performance-information acquisition unit 124 acquires, from theexternal apparatus such as the management server 200 through the network300, performance information regarding the muffler from Company A theattribute of which is judged to be included in the real data image 21A.For example, information indicating a capacity of 296 ps (pferde starke(horsepower)) and a torque of 38 kgm is acquired as the performanceinformation regarding the muffler from Company A. The virtual-imageincorporation unit 125 acquires a virtual data image associated with“Muffler from Company A” from the virtual-image storage unit 123. Thevirtual-image incorporation unit 125 incorporates, into the racing game,the performance information and the virtual data image in associationwith each other.

FIG. 11B is a diagram illustrating an example of a data image 21C thatis displayed on the display 105 after being incorporated into the racinggame. In the example illustrated in FIG. 11B, a virtual data image 21Dassociated with “Muffler from Company A” is displayed on the display 105as the image of the muffler captured by the user, with the motor cycleprovided with the virtual data image 21D. The user may use, in theracing game, the muffler from Company A having the performance indicatedby the performance information.

In this exemplary embodiment as described above, the operation terminal100 identifies the attribute of an object from a real data imagecaptured by using the imaging unit 107 and incorporates, into thevirtual space, a virtual data image and performance information thatcorrespond to the object the attribute of which is identified. Theincorporation of the virtual data image and the performance informationinto the virtual space enables the user to use, in the virtual space,information regarding the object in the subject the image of which iscaptured.

Note that although the virtual-image incorporation unit 125 acquires theperformance information from the external apparatus such as themanagement server 200 through the network 300 in this exemplaryembodiment, the performance information may be included in thevirtual-image DB. In this case, the virtual-image DB illustrated in FIG.9 also stores therein pieces of performance information on aper-object-attribute basis, such as for “Muffler from Company A”,“Muffler from Company B”, and “Tire from Company C”.

In addition, although the real-image acquisition unit 121 acquires animage actually captured using the imaging unit 107 in this exemplaryembodiment, the configuration is not limited to such a configuration.For example, the real-image acquisition unit 121 may acquire a dataimage captured by using another imaging unit from the management server200 through the network 300 or from a recording medium such as a compactdisc read-only memory (CD-ROM).

Exemplary Embodiment 3 Functional Configuration of Operation Terminal

The functional configuration of an operation terminal 100 according toExemplary Embodiment 3 will be described. In Exemplary Embodiment 2, theoperation terminal 100 identifies the attribute of an object from a realdata image and executes the process for incorporating, into the virtualspace, performance information regarding the object the attribute ofwhich is identified. In Exemplary Embodiment 3, the operation terminal100 identifies the attribute of an object from a sound (sound data) andexecutes a process for incorporating, into the virtual space,performance information regarding the object the attribute of which isidentified.

FIG. 12 is a block diagram illustrating an example functionalconfiguration of the operation terminal 100 according to ExemplaryEmbodiment 3. The operation terminal 100 according to this exemplaryembodiment includes a sound collection unit 131, an objectidentification unit 132, a virtual-image storage unit 133, aperformance-information acquisition unit 134, and a virtual-imageincorporation unit 135.

The sound collection unit 131 collects a sound (sound data) detected inthe real space by the sound detection unit 108 and records the collectedsound in a memory such as the magnetic disk device 103.

The object identification unit 132 identifies the attribute of an objecton the basis of the sound of a predetermined object (a sound emittedfrom the predetermined object) included in the sound collected by thesound collection unit 131. An existing technique is usable for theidentification. For example, the user has prepared sounds for variousobjects. The object identification unit 132 performs processing such aspattern matching on, for example, the waveform of the sound collected bythe sound collection unit 131 and thereby identifies the attribute ofthe object.

The virtual-image storage unit 133 stores therein a virtual-image DBthat is a database in which virtual data images corresponding to piecesof image data to be used in virtual space. In the virtual-image DB,virtual data images corresponding to predetermined objects are specifiedon a per-predetermined-object-attribute basis. Further on aper-predetermined-object-attribute basis, sounds associated with therespective objects are also stored. More specifically, for example, if aracing game is played in the virtual space, the virtual-image DB storestherein virtual data images on a per-automobile-attribute basis, such asfor “Automobile from Company A” and “Automobile from Company B”. On aper-automobile-attribute basis, sounds such as an engine sound and adoor closing sound of the automobiles are also stored as soundsassociated with the respective automobiles.

The performance-information acquisition unit 134 acquires performanceinformation indicating the performance of the object the attribute ofwhich is identified by the object identification unit 132. Theperformance-information acquisition unit 134 acquires the objectperformance information from an external apparatus such as themanagement server 200 through the network 300. More specifically, theperformance-information acquisition unit 134 searches for the object,for example, through the Internet and thereby acquires the objectperformance information.

The virtual-image incorporation unit 135 executes a process forincorporating, into the virtual space, the performance informationregarding the object that is acquired by the performance-informationacquisition unit 134 and a virtual data image corresponding to theobject. The virtual-image incorporation unit 135 acquires, from thevirtual-image storage unit 133, the virtual data image corresponding tothe object the attribute of which is identified by the objectidentification unit 132, in other words, the virtual data imageassociated with the attribute of the object. The virtual-imageincorporation unit 135 incorporates, into the virtual space, theperformance information and the virtual data image in association witheach other. The incorporation into the virtual space causes the virtualdata image to function as an image of the object having the performanceindicated by the performance information in the virtual space.

More specifically, for example, if a racing game is to be played, theobject identification unit 132 identifies “Automobile from Company A” asthe attribute of the object. In this case, the performance-informationacquisition unit 134 searches for the automobile from Company A throughthe Internet and acquires the performance information regarding theautomobile from Company A. The virtual-image incorporation unit 135acquires a virtual data image corresponding to the automobile fromCompany A from the virtual-image storage unit 133. The virtual-imageincorporation unit 135 incorporates, into the racing game, theperformance information and the virtual data image in association witheach other. The incorporation into the racing game enables the user touse, in the racing game, the automobile from Company A having theperformance indicated by the performance information.

In this exemplary embodiment, the performance-information acquisitionunit 134 is used as an example of a performance-information acquisitionunit. The virtual-image incorporation unit 135 is used as an example ofeach of the identification unit and the incorporation unit.

Note that like the functional units included in the operation terminal100 illustrated in FIG. 3, the functional units included in theoperation terminal 100 illustrated in FIG. 12 are implemented incooperation between the software and hardware resources. Specifically,to implement the operation terminal 100 by using the hardwareconfiguration illustrated in FIG. 2, the OS and application programsstored in the magnetic disk device 103 are loaded in the main memory 102and run by the CPU 101, and the functions such as the sound collectionunit 131, the object identification unit 132, theperformance-information acquisition unit 134, and the virtual-imageincorporation unit 135 are thereby implemented. The virtual-imagestorage unit 133 is implemented by a memory such as the magnetic diskdevice 103.

Virtual-Image DB

The virtual-image DB will be described. FIG. 13 is a table illustratingan example of the virtual-image DB according to Exemplary Embodiment 3.

As “Object attribute”, attributes of the predetermined objects areillustrated. In the example in FIG. 13, “Automobile from Company A”,“Automobile from Company B”, and “Automobile from Company C” areillustrated as automobiles. As “Sound”, sounds associated with therespective objects are illustrated. For example, an engine sound of theautomobile from Company A and a door closing sound of the automobilefrom Company A are illustrated as sounds of the automobile from CompanyA. Although characters are used in the example in FIG. 13 such as“Engine sound” and “Door closing sound”, pieces of data regarding soundssuch an actual engine sound and an actual door closing sound are storedin the virtual-image DB. In addition, as “Virtual data image”, dataimages for a case where objects are used in the virtual space such asthe racing game.

Steps of Process Performed by Operation Terminal

Steps of a process performed by the operation terminal 100 according tothis exemplary embodiment will be described. FIG. 14 is a flowchartillustrating an example of the steps of the process performed by theoperation terminal 100 according to Exemplary Embodiment 3.

When the user operates the operation terminal 100 and when the sounddetection unit 108 detects a sound, the sound collection unit 131collects the sound and records the sound in the memory (step S401). Theobject identification unit 132 judges whether the sound collected by thesound collection unit 131 has an attribute of a predetermined object onthe basis of the information stored in the virtual-image storage unit133 (step S402). If the object identification unit 132 does not judgethat the sound has an attribute of a predetermined object (No in stepS402), the process is terminated.

In contrast, if the object identification unit 132 judges that the soundhas an attribute of a predetermined object (Yes in step S402), theperformance-information acquisition unit 134 acquires performanceinformation corresponding to the predetermined object (step S403). Thevirtual-image incorporation unit 135 acquires a virtual data imageassociated with the predetermined object from the virtual-image storageunit 133 (step S404). The virtual-image incorporation unit 135incorporates, into the virtual space, the performance information andthe virtual data image in association with each other (step S405). Theprocess is then terminated.

A Series of Steps Performed by Operation Terminal

A series of steps performed by the operation terminal 100 according tothis exemplary embodiment will be described. FIGS. 15A and 15B arediagrams for explaining an example of the series of steps performed bythe operation terminal 100 according to Exemplary Embodiment 3. A casewhere a racing game is played in virtual space will be described as anexample.

FIG. 15A is a diagram for explaining an example of a sound acquired bythe sound detection unit 108. For example, the user actually runs theengine of an automobile, and the sound detection unit 108 detects anengine sound. As the result, the sound collection unit 131 collects thesound detected by the sound detection unit 108.

After the sound collection unit 131 collects the engine sound, theobject identification unit 132 judges where the collected engine soundhas an attribute of a predetermined object on the basis of theinformation stored in the virtual-image storage unit 133. If thecollected engine sound is the same (or similar to) as the engine soundof, for example, the automobile from Company A illustrated in FIG. 13,the object identification unit 132 judges that the collected enginesound has the attribute of the automobile from Company A on the basis ofpattern matching or the like.

The performance-information acquisition unit 134 acquires, from theexternal apparatus such as the management server 200 through the network300, performance information corresponding to the automobile fromCompany A the attribute of which is judged to be included in thecollected engine sound. For example, information indicating adisplacement of 5000 cc, a capacity of 280 ps, and a maximum speed of200 km/h is acquired as the performance information regarding theautomobile from Company A. The virtual-image incorporation unit 135acquires a virtual data image associated with the automobile fromCompany A from the virtual-image storage unit 133. The virtual-imageincorporation unit 135 incorporates, into the racing game, theperformance information and the virtual data image in association witheach other.

FIG. 15B is a diagram illustrating an example of a data image 31Aincorporated into the racing game and then displayed on the display 105.In the example illustrated in FIG. 15B, a virtual data image 31B of theautomobile from Company A is displayed on the display 105 as a dataimage associated with the collected engine sound. The performanceinformation indicates a displacement of 5000 cc, a capacity of 280 ps,and a maximum speed of 200 km/h. The user may use, in the racing game,the automobile from Company A having the performance indicated by theperformance information.

In this exemplary embodiment as described above, the operation terminal100 identifies the attribute of an object from the sound collected bythe sound collection unit 131 and incorporates, into the virtual space,a virtual data image and performance information that correspond to theobject the attribute of which is identified. The incorporation of thevirtual data image and the performance information into the virtualspace enables the user to use, in the virtual space, informationregarding the object associated with the collected sound.

Note that although the virtual-image incorporation unit 135 acquires theperformance information from the external apparatus such as themanagement server 200 through the network 300 in this exemplaryembodiment, the performance information may be included in thevirtual-image DB as in Exemplary Embodiment 2. In this case, thevirtual-image DB illustrated in FIG. 13 also stores therein pieces ofperformance information on a per-object-attribute basis, such as for“Automobile from Company A”, “Automobile from Company B”, and“Automobile from Company C”.

Although the sound collection unit 131 collects the sound actuallydetected by the sound detection unit 108 in this exemplary embodiment,the configuration is not limited to such a configuration. For example,the sound collection unit 131 may collect, as sound data, a soundreceived from the management server 200 through the network 300 or asound provided in such a manner as to be stored in a recording mediumsuch as a CD-ROM.

The process performed by each operation terminal 100 according to acorresponding one of Exemplary Embodiments 1 to 3 does not have to beexecuted by only the operation terminal 100 and may be shared withanother apparatus. For example, in Exemplary Embodiment 1, the operationterminal 100 may have the functions of the real-image acquisition unit111 and the combining unit 116, and the management server 200 may havethe functions of the region division unit 112, the region-attributeidentification unit 113, the virtual-image storage unit 114, and thevirtual-image acquisition unit 115. In this case, for example, theoperation terminal 100 acquires a real data image and thereaftertransmits the acquired real data image to the management server 200. Themanagement server 200 identifies a virtual data image to be combinedwith the real data image and transmits the identified virtual data imageto the operation terminal 100. This causes the operation terminal 100 tocombine the real data image with the virtual data image and therebygenerate a composite data image.

Further, note that programs for implementing the exemplary embodimentsof the invention may be provided not only through a communication mediumbut also in such a manner as to be stored in a recording medium such asa CD-ROM.

Note that various exemplary embodiments and modifications describedabove as Exemplary Embodiments 1 to 3 may be combined. That is, toimplement the operation terminal 100, for example, all of or two ofExemplary Embodiments 1 to 3 may be combined together. Alternatively,for example, one of Exemplary Embodiments 1 to 3 may be used toimplement the operation terminal 100.

The present disclosure is not limited to the exemplary embodimentsdescribed above and may be implemented in various forms withoutdeparting from the spirit of the disclosure.

The foregoing description of the exemplary embodiments of the presentinvention has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit theinvention to the precise forms disclosed. Obviously, many modificationsand variations will be apparent to practitioners skilled in the art. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, therebyenabling others skilled in the art to understand the invention forvarious embodiments and with the various modifications as are suited tothe particular use contemplated. It is intended that the scope of theinvention be defined by the following claims and their equivalents.

1. An information processing system comprising: a processor programmedto acquire a real image captured in a real space, divide the real imageinto a plurality of regions, identify an attribute of each of theplurality of regions, select one or more regions of the plurality ofregions in accordance with a predetermined rule, identify one or moreprestored virtual images stored in association with the identifiedattribute of the selected one or more regions, and combine the one ormore prestored virtual images with the acquired real image such that theone or more prestored virtual images are fixed in the selected one ormore regions, to thereby generate a composite image.
 2. The informationprocessing system according to claim 1, further comprising: a displaythat displays the composite.
 3. The information processing systemaccording to claim 1, further comprising: a memory that stores anassociation between the prestored virtual images and the attributes ofthe plurality of regions.
 4. The information processing system accordingto claim 2, further comprising: a memory that stores an associationbetween the prestored virtual images and the attributes of the pluralityof regions.
 5. (canceled)
 6. An information processing systemcomprising: a processor programmed to identify a virtual imagecorresponding to an object that is present in a real space, the virtualimage being identified based on a sound emitted from the object, acquireperformance information indicating performance of the object, andincorporate, into a virtual space, the acquired performance informationand the virtual image corresponding to the object in association witheach other.
 7. An information processing system comprising: a processorprogrammed to acquires a real image captured in a real space, acquire apredetermined virtual image from a virtual image database, identify aregion in the acquired real image where the predetermined virtual imageis to be combined with the acquired real image, the region beingidentified based on an attribute of the predetermined virtual image, andcombine the predetermined virtual image with the acquired real imagesuch that the predetermined virtual image is fixed in the identifiedregion, to thereby generate a composite image.
 8. The informationprocessing system according to claim 7, wherein the processor is furtherprogrammed to determine whether the acquired real image contains anyregion corresponding to the attribute of the predetermined virtualimage, and in response to a determination that the acquired real imagecontains no region corresponding to the attribute, the processor doesnot generate a composite image using the predetermined virtual image andthe acquired real image.